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Video camera with colour-based target tracking system. 



(57) in a video camera system, a measuring frame 
is set on an object in order not to be affected by 
intensity and direction of the light source. A 
display screen is divided into divided areas (AR) 
and, regarding each divided area, a lightness 
divided area in which a brightness signal has 
the prescribed luminance, and a skin color 
divided area in which a hue signal is skin color 
part, are extracted. Depending upon the extrac- 
tion result, the position on the screen of the 
object is detected. The measuring frame which 
would not be affected by intensity and direction 
of the light source can then be set utilizing the 
hue signal. 
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This invention relates to a video camera system 
which can be applied in automonous target tracking 
apparatus in which the field of view of a video camera 
can track the center of an object such as a human 
face. 5 

Heretofore in video camera autonomous target 
tracking apparatus, a first automonous target tracking 
method has been proposed in which the peak value 
of a high frequency element of a brightness signal in 
the measuring frame is memorized and its movement 10 
is automatically tracked as the feature of the target. 

A second automonous target tracking method 
has been proposed in which, regarding brightness 
signals of the front and back fields in the measuring 
frame, a motion vector is formed by taking the key 15 
point matching and it is assumed that the motion vec- 
tor in the measuring frame is the movement of the tar- 
get. 

The first automonous tracking method has an ad- 
vantage in that the general construction is simple, 20 
but, for example in the case where there is a man in 
the measuring frame with a tree in the background, 
the man cannot be tracked automatically since the 
high frequency element of the tree is the biggest peak 
at this point 25 

Furthermore, since basically the first automon- 
ous tracking method utilizes peak signals, this meth- 
od is easily affected by noise. As a result, there is a 
possibility that automatic tracking cannot be per- 
formed in a photographic environment with low bright- 30 
ness. Also, it is possible that an object with poor con- 
trast cannot automatically be tracked since basically 
the high frequency element is extracted. 

Using the second automonous tracking method, 
it is difficult to distinguish whether the calculated mo- 35 
tion vector is caused by unintentional movement of 
the video camera or by the movement of the object, 
and in practice there is a possibility of the system 
functioning erroneously. 

According to one aspect of the invention there is 40 
provided a video camera system comprising: an area 
dividing means (16, SP1 ) for dividing a display screen 
PIC to be shown by picture output signal S1 outputted 
from a lens block unit 1 into divided areas AR of the 
prescribed numbers 32 x 32, 16 x 16, a skin color div- 45 
ided area extracting means (16, SP1) for extracting 
skin color divided areas ARB in areas AR wherein hue 
signal HUE is skin color part, a lightness area extract- 
ing means (16, SP1) for extracting lightness divided 
area ARA in whole divided areas AR wherein bright- 50 
ness signal Y has the prescribed lightness, an ab- 
sorbing means (16, SP2) for forming skin divided area 
groups FC1, FC2 and lightness divided area groups 
HR1, HR2 by absorbing blank areas around the skin 
divided area ARB and lightness divided area ARA, an 55 
adjacent divided area group extracting means (16, 
SP5) for extracting a set of adjacent skin divided area 
group FC1 and lightness divided area group HR1 in 



skin divided area groups FC1 , FC2 and lightness div- 
ided area groups HR1, HR2, and a measuring frame 
setting means (16, SP9) for setting said measuring 
frame PIC in order that a set of the skin divided area 
group FC1 and the lightness divided area group HR1 
are included in the measuring frame PIC. 

When blank areas AR around the skin divided 
area ARB and the lightness divided area ARA are ab- 
sorbed, even though blank areas occur in the skin div- 
ided area ARB and the lightness divided area ARA af- 
fected by the intensity and/or direction of the light 
source, the skin divided area and the lightness divid- 
ed area can be extracted properly because they are 
absorbed in skin divided area groups FC1, FC2 and 
lightness area groups HR1, HR2, and simultaneously, 
the measuring frame for the object can be set with 
certainty because the measuring frame is set based 
on the adjacent set FC1 , HR1 in said skin divided area 
groups FC1, FC2 and the lightness divided area 
groups HR1, HR2. 

Respective further aspects of the invention are 
as set forth in claims 1 , 4, 6, 7, 8 and 9. 

In embodiments of the invention, since the image 
information of each pixel which constitutes the dis- 
play screen is divided into the prescribed number of 
small divided areas, and the measuring frame is set 
by judging the position and size of the skin color area 
based on the hue signal of each small divided area, 
a video camera system which is capable of setting the 
measuring frame on the object with high accuracy can 
be easily obtained. 

Embodiments of the invention provide video cam- 
era automonous tracking apparatus which is in gen- 
eral unaffected by the intensity and direction of the 
light source and, simultaneously, which is in general 
unaffected by the background. 

Emodiments of the invention will now be descri- 
bed, byway of example, with reference to the accom- 
panying drawings in which: 

Fig. 1 is a block diagram showing the construc- 
tion of video camera automonous target tracking 
apparatus applicable to the video camera sys- 
tem; 

Fig. 2 is a schematic diagram illustrating a HLS 
color coordinate system which expresses the vis- 
ual stimulation; 

Fig. 3 is a schematic diagram showing a display 
screen PIC wherein the image information can be 
obtained by dividing into small areas; 
Fig. 4 is a flow chart showing the tracking proc- 
ess procedure; 

Fig. 5 is a flow chart showing the tracking proc- 
ess procedure continued from Fig. 4; 
Fig. 6 is a schematic diagram showing the display 
screen PIC having hair areas and skin color areas 
detected; 

Fig. 7 is a schematic diagram showing the display 
screen PIC absorbed and/or combined; and 
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Fig. 8 is a block diagram showing a further em- 
bodiment of video camera target tracking appara- 
tus. 

In Fig. 1, ATT generally shows an autonomous 
target tracking apparatus of the video camera system, 
and the autonomous target tracker aims at a human 
face as its target Thus, it is applicable as a means to 
make a lens block of the video camera system con- 
stituting the Autonomous Target Tracker with Human 
Recognition, ATT-HR system for automatically track- 
ing function. 

The autonomous target tracking apparatus ATT 
receives an image light LA which comes from the hu- 
man face model as the object at a charge coupled de- 
vice 4 (CCD) through a lens 2 of the lens block unit 1 
and an iris 3 and delivers an image output signal S1 
which shows the target image to a signal separa- 
tion/automatic gain adjusting circuit 5. 

The signal separation/automatic gain adjusting 
circuit sample holds the image output signal S1 f 
which is gain controlled in order to have the prescri- 
bed gain by the control signal from an auto iris (AE) 
system (not shown) and thus obtained image output 
signal S2 will be supplied to a digital camera process- 
ing circuit 7 through an analog-to-digital converter 6. 

The digital camera processing circuit 7 forms 
brightness signal Y, chrominance signal C and color 
difference signals R-Y and B-Y according to the im- 
age output signal S2, and outputs the brightness sig- 
nal Y and chrominance signal C through a digital-to- 
analog converter 8 as video signal S3. 

Furthermore, the digital camera processing cir- 
cuit 7 supplies brightness signal Y and color differ- 
ence signals R-Y and B-Y as target tracking detection 
signal S4 to a tracking control circuit 11 , which gener- 
ates tracking control signal S5 for a panning driving 
motor 12 and a tilting driving motor 13 which are 
equipped in the lens block unit 1 in accordance with 
the target tracking detection signal S4. 

First of all, the tracking control circuit unit 11 sup- 
plies color difference signals R-Y and B-Y to a satur- 
ation/hue detection circuit 14 and forms the hue sig- 
nal HUE and the saturation signal SAT and memoriz- 
es these with brightness signal Y in an image memory 
1 5 constituted by, such as field memory, per each pic- 
ture element unit. 

The saturation/hue detection circuit 14 forms hue 
signal HUE and saturation signal SAT by rectangular 
coordinate to polar coordinate converting color differ- 
ence signals R-Y and B-Y, and recognizes the human 
face model as an object based on the visual stimula- 
tion which human being can perceive by means of 
brightness signal Y, hue signal HUE and saturation 
signal SAT. 

In this connection, as shown in Fig. 2, the visual 
stimulation which human beings can generally per- 
ceive, can be expressed in L axis and color coordinate 
system which contains SH plane perpendicularly in- 



tersecting to L axis, as it is called "HLS system 0 . Laxis 
shows "lightness" and this is equivalent to brightness 
signal Y. SH plane is expressed by po\ar coordinate 
which intersects perpendicularly to L axis. In the SH 

5 plane, S shows "saturation" and is expressed by the 
distance from L axis. Also, H stands for "hue" and hue 
is expressed by an angle at the tone when the direc- 
tion of hue signal B-Y is 0 [°]. 

The cubic factor of this HLS is represented that, 

10 as the light source becomes lighter, the color coordin- 
ate, i.e., SH plane, moves upward along the L axis, 
and all colors become white. At this point the satur- 
ation S gradually decreases. On the other hand, if the 
light source becomes darker, the color coordinate, 

15 i.e., SH plane, falls down along the L axis, and all col- 
ors become black. At this point, the saturation S also 
decreases. 

As described above, the saturation S and the 
brightness Y are easily affected by the lightness of 
20 the light source in accordance with the characteristic 
of HLS color coordinate system; and therefore, it is 
difficult to say that they are the optimal parameters 
to express the quantity of features of the object On 
the other hand, it is obvious that the hue H is the 
25 quantity which is not easily affected by the light 
source as a parameter to express the quantity of in- 
herent features of the object 

However, in the case where color of the object is 
in the neighborhood of L axis, i.e., in the case of 
30 white, black or gray, the signal of hue H has no mean- 
ings as the information, and in the worst case, in spite 
of the fact that the picture with poor S/N is white, it 
seems possible to have various hue H vectors. 

In utilizing these features of HLS color coordinate 
35 system, the tracking control circuit 11 extracts the fea- 
ture of the human face model as the object and when 
the feature changes, the panning driving motor 12 
and tilting driving motor 13 will be actuated in order to 
follow the changes and as a result, video signal which 
40 tracks the movement of the object as video signal S3 
will be obtained. 

More specifically, target tracking detection signal 
S4 memorized in an image memory 16 supplies ad- 
dress signal S7 to divide the display screen PIC 
45 formed in the image memory 15 into divided areas AR 
of the prescribed size in accordance with xy rectan- 
gular coordinates (x, y). as shown in Fig. 3, by giving 
block defining signal S6 to be outputted from the mi- 
croprocessor composed tracking signal processing 
so circuit 16 to an address generation circuit 17. 

Thus, the data of each picture element which con- 
stitute the display screen PIC of the image memory 
16 will be read out in every divided area AR and will 
be evaluated as an image information in each divided 

55 area AR. 

In the case of this embodiment, the display 
screen PIC is divided into 32 (or 16) small divided 
areas AR in the directions of x and y respectively, and 
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thus by designating the coordinates of rectangular 
coordinate (x, y) as x = i, y = j, regarding 32 x 32 (or 
16 x 16) small divided areas AR f the image informa- 
tion I (x = i, y = j) of small divided areas to be desig- 
nated can be read out. 

With this arrangement whereas the hue signal 
HUE element will be given to a skin color detection cir- 
cuit 19 through a gate circuit 18 in the image informa- 
tion I (x = i, y = j) to be read out in every small divided 
area from the image memory 15, the brightness sig- 
nal Y element will be given directly to a hair color de- 
tection circuit 20. 

The skin color detection circuit 19 detects the im- 
age part of skin of the human face model and when 
the incoming hue signal HUE element is within the 
range of prescribed skin color, it gives "+1" count out- 
put S8 to a skin color pixel number counter 21 as the 
image of said picture element is skin color. 

In practice, the dynamic range of hue signal HUE 
takes the value in the range of an angle 0 - 350 [°]. 

On the other hand, the hair color detection circuit 
20 detects hair areas in the image part of the human 
face model and in the case of this embodiment, the 
dynamic range of brightness signal Y is expressed by 
the value 0 - 255 (8 bits) and when the brightness sig- 
nal of each picture element is below the value 50, it 
defines as black and judges that said picture element 
exists in the hair area and outputs count output 
S9 to a hair pixel number counter 22. 

With this arrangement, the number of pixels hav- 
ing information on skin color and haircolor respective- 
ly in pixels contained in each divided area AR, will be 
counted in the skin color pixel number counter 21 and 
the hair pixel number counter 22. 

The counting result of each divided area AR of the 
skin color pixel number counter 21 and the hair pixel 
number counter 22 will be compared with judgement 
reference signals S10 and S11 to be given from the 
tracking signal processing circuit 16 at comparators 
23 and 24. 

At this point, judgement reference signals S10 
and S11 are set to the value to determine whet her the 
counting result of skin color pixel number counter 21 
and hair pixel number counter 22 will be judges as 
prospective skin color area or prospective hair area or 
not for said divided areas AR, and thus, comparators 
23 and 24 transmit prospective skin color area detec- 
tion information S12 and prospective hair area detec- 
tion information S13 to the tracking signal processing 
circuit 16. 

In the case of this embodiment, a hue noise gate 
signal forming circuit 25 composed of comparator is 
equipped for the gate circuit 18 and it compares sat- 
uration signal SA1 to be read out in every pixel from 
the image memory 15 with noise judgement signal 
S14 to be outputted from tracking signal processing 
circuit 16, and by giving gate signal S15 which oper- 
ates the gate circuit to close when saturation signal 



SAT is under the prescribed level, it is so arranged 
that the hue signal HUE element of said pixel will not 
be inputted to the skill color detection circuit 19. 
In this connection, in the case where the hue sig- 

5 nal HUE detected at the saturation/hue detection cir- 
cuit 14 is in the neighborhood of the L axis (Fig. 2) and 
this means that said hue signal HUE has small satur- 
ation and for fear that it is buried in noises and has no 
meanings as information, these meaningless hue sig- 

10 nals will be eliminated at the gate circuit 18. 

According to the foregoing construction, the 
tracking signal processing circuit 16 judges the fea- 
ture of an image on the display screen PIC from the 
prospective skin color area detection information S12 

15 and the prospective hair area detection information 
S13 to be obtained based on brightness signal Y, hue 
signal HUE and saturation signal SAT of each pixel 
contained in the image memory 1 5 and delivers track- 
ing control signal S5 so as the center of gravity of hu- 

20 man face model comes in the center of the measuring 
frame constantly, to the panning driving motor 12 and 
the tilting driving motor 13. 

In the case of this embodiment, supposing that 
the display screen PIC is generally one measuring 

25 frame, it is so arranged that the centroid of a face 
model would come in the center of display screen 
PIC. 

When the tracking signal processing circuit 16 
enters in the tracking processing procedure RT1 of 

30 Fig. 4, first of all, it sets the field number FN to the ini- 
tial value, FN - 1, and moves to step SP1 and divides 
the display screen PIC into 32 x 32 divided areas AR, 
and then obtains counting results of the skin color pix- 
el number counter 21 and the hair pixel number coun- 

35 ter 22 based on said detection result of the skin color 
detection circuit 1 9 and the hair detection circuit 20 for 
pixels contained in each divided area AR. In utilizing 
these the tracking signal processing circuit 16 takes 
into the prospective skin color area detection infbrma- 

40 tion S12 when said divided area is the prospective 
skin color area and at the same time it takes into the 
prospective hair area detection information S13 when 
the divided area AR is the prospective hair area. 
When the above procedure has been terminated 

45 at the step SP1 . the tracking signal processing circuit 
16 appoints the field number to be extracted and 
processed next time by incrementing the field number 
FN to FN = FN + 1. 

With this arrangement the procedure of step SP1 

so has been terminated, and the tracking signal proc- 
essing circuit 16 is able to obtain the distribution con- 
dition of divided areas AR judged as the prospective 
hair divided areas ARA and prospective skin color div- 
ided areas ARB as shown in Fig. 3 or Fig. 6. 

55 At this point, as shown in Fig. 3, if the prospective 

hair divided areas ARA and the prospective skin color 
divided areas ARB are adjacent each other without 
space and moreover, the extraction result showing 
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that there is no space between the prospective hair 
divided areas ARA and the prospective skin color div- 
ided areas ARB is obtained, it means that the infor- 
mation to conform to the actual condition of human 
face model as an object has been obtained. 

However, since in ordinary circumstances the di- 
rection and reflection manner of the light source for 
the object are not uniform, there occur blank areas in 
the prospective hair divided areas ARA and prospec- 
tive skin color divided areas ARB as shown in Fig. 6, 
and moreover, there are many cases where extracting 
results show that there are blank areas in the group 
of prospective hair divided areas ARA and the group 
of prospective skin color divided areas ARB. 

Then, the tracking signal processing circuit 1 6 ab- 
sorbs said discontinuous areas according to (hair 
area absorbing rule 1) and (skin color area absorbing 
rule 2) at the following step SP2. 

The "absorption rule 1" of discontinuous areas is 
"If I (i + P, j + q) - hair area, (p, q = +1 or -1 or 0, pro- 
vided excepting p = q = 0), then absorb I (i, j) as the 
prospective hair area". 

This "absorption rule 1" of discontinuous areas 
means that in the address (x = i, y = j) area the image 
information I (i, j) of said designated divided areas will 
be absorbed in the prospective hair divided areas 
ARA in the case where there are prospective hair div- 
ided areas ARA in the neighboring areas centering 
around this divided areas AR. 

For example, in Fig. 6 in the case where the blank 
divided area (x = 6, y = 5) is designated as an appoint- 
ed area, the hair divided area (x = 6, y = 4) is included 
as prospective hair divided area ARA1 in divided 
areas AR, i.e., (x = 5, y = 4), (x = 6, y = 4), (x = 7, Y = 
4), (x = 7, Y = 5), (x = 7, y = 6). (x = 6, y = 6), (x = 5, 
y = 6) and (x = 5, y = 6) which surround this blank div- 
ided area, and thus this blank divided area (x = 6, y 
= 5) is absorbed in a group of prospective hair divided 
areas ARA1 by the "absorption rule 1". 

Also, the absorption rule of discontinuous area 
will be applied after applying the "absorption rule 2", 
and its content is as follows: If I (i + p, j + q) = prospec- 
tive skin color divided area; (p, q = +1 or -1 or 0, pro- 
vided, excepting p = q = 0), then I (i, j) is absorbed as 
prospective skin color divided area. 

The "absorption rule 2" of this discontinuous area 
is the same as described above regarding prospec- 
tive hair divided area ARA. If there are prospective 
skin color areas in the neighbor areas concerning the 
designated divided area (x = i, y = j), the image infor- 
mation i (x = i, y = j) of said designated divided area 
(x = i, y = j) will be absorbed as prospective skin color 
divided area ARB. 

The "absorption rule 2" of this discontinuous area 
is the same as described above regarding prospec- 
tive hair divided area ARA. If there are prospective 
skin color areas in the neighbor areas concerning the 
designated divided area (x = i, y = j), the image infor- 



mation i (x = i, y = j) of this designated divided area 
( X = i ( y = j) will be absorbed as prospective skin color 
divided area ARB. 

For example, in Fig. 6 in the case where the blank 
5 divided area (x = 6, y = 6) is appointed as a designated 
area, since the skin color area (x = 7, y = 7) is included 
as prospective skin color divided area ARB1 in divid- 
ed areas AR surrounding this blank divided area (x = 
6,Y = 6),i.e., (x = 5,y = 5),(x = 6,y = 5), (x = 7,y = 
10 5),(x = 7,y = 6),(x = 7.y=7),(x = 6.y = 7),(x = 5,y 
= 7) and (x = 5, y = 6), this blank divided area (x - 6, 
y = 6) is absorbed in a group of prospective skin color 
areas ARB1 by the "absorption rule 2". 

Accordingly, the absorption process of discontin- 
15 uous area of the step SP2 is terminated and as shown 
in Fig. 6, since there were blank areas in the group of 
prospective hair divided areas ARA1, the group of 
prospective hair divided areas ARA1 was discontinu- 
ous, however, this blank areas can be filled in by the 
20 absorbed prospective skin color divided area ARB2 
as shown in Fig. 7. 

The tracking signal processing circuit 16 pro- 
ceeds to the step SP3 when the process of step SP2 
is terminated and defines the determination of hair 
25 area and skin color area by the "hair area determina- 
tion rule 1" and the "skin color area determination rule 
2". 

The "hair area determination rule 1" is "If I (i, j) - 
prospective hair area and I (i + p, j + q) = prospective 

30 hair area, (here p, q = +1 or -1 or 0, provided, except- 
ing p = q = 0), then I (i, j), I (i + p. j + q) will be combined 
as hair area. 

This "hair area determination rule 1" means that 
in the address (x = i, y = j) area, the image information 

35 (i, j) of said designated area (x = i, y = j) will be com- 
bined in the hair divided area group HR, in the case 
where there are prospective hair divided areas in the 
neighboring divided areas centering around said div- 
ided areas. 

40 Also, the "skin color area determination rule 2" 

combines I (i, j), I (i + p, j + <0 " skin color area. "If I 
(i, j) is prospective skin color area and I (i + p, j + q) 
is prospective skin color area, (here p, q = +1 or -1 or 
0, provided, excepting p = q = 0)°. 
45 This "skin color area determination rule 2" is the 

same as described above regarding the prospective 
hair divided area, the image information (x = i, y = j) 
and the image information of neighboring area where 
there exists the prospective skin color divided area 
so group FC if there are prospective skin color areas in 
the neighboring areas regarding the designated div- 
ided area (x = i, y = j). 

Then, for example, as described above, if the 
"hair area determination rule 1" is applied to the area 
55 (x = 6, y = 5) absorbed in the hair divided area (Fig. 
6) by the "hair area absorption rule 1 M , hair divided 
areas (x = 5, y = 5), (x = 5, y = 4), (x = 6, y = 4), (x = 
7 f y = 4) and (x = 7, y = 5) are included as prospective 
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hair divided areas in divided areas around the divided 
area (x = 6, y = 5), i.e., (x = 5, y = 4), (x = 6, y = 4), (x 
= 7,y = 4),(x = 7,y = 5), (x = 7,y = 6) t (x= 6,y = 6) t 
(x = 5, y = 6) and (x = 5, y = 5), and areas (x = 5, y = 

4) , (x = 6,y = 4), (x = 7,y = 4), <x = 7,x = 5), <x = 6,y 
= 5) and (x = 5, y = 5) will be combined as the hair div- 
ided area group FC1. 

Moreover, for example, as described above, if the 
"skin color area determination rule 2" is applied to the 
divided area (x = 6, y = 6) absorbed in the skin color 
divided area by the "skin color area absorption rule 2", 
areas (x = 6, y = 6), (x = 7, y = 6), (x = 7, y = 7) and 
(x = 6, y = 7) wBI be combined as the skin color divided 
area group FC1 because skin color divided areas (x 
= 7, y = 6), (x = 7, y = 7) and (x = 6, y = 7) are included 
as prospective skin color areas in divided areas 
around the divided area (x = 6, y = 6), i.e., (x = 5, y = 

5) , (x = 6,y = 5), (x = 7,y = 5), (x = 7, y = 6), (x = 7, y 
= 7), (x = 6, y = 7), (x = 5, y = 7) and (x = 5, y = 6). 

Thus, as shown in Fig. 7, divided area groups 
HR1 and HR2 in the display screen PIC are combined 
as the hair area and divided area groups FC1 and 
FC2 are combined as the skin color area at the steps 
SP2 and SP3 by absorbing discontinuous divided 
areas and by combining hair divided area and skin 
color divided area. 

Then, the tracking signal processing circuit 16 
proceeds to the step SP4 of Fig. 5 and calculates the 
sizes Area_Hair and Area_Skin of combined hair 
areas HR1 and HR2, and skin color areas FC1 and 
FC2. 

In the case of display screen PIC, as shown in 
Fig. 7, the size Area_Hair of the hair area HR1 is cal- 
culated as the total number of areas contained in the 
hair area HR1, i.e. Area_Hair = 45. In the same man- 
ner, the size Area_Hair of the hair area group HR2 is 
calculated as the Area_Hair =14, the size Area_Skin 
of skin color area group FC1 is calculated as 
Area_Skin = 66 and the size Area_Skin of skin color 
area group FC2 is calculated as Area_Skin = 10. 

Then, the tracking signal processing circuit 16 
proceeds to the step SP5 and judges the process to 
determine the face area under the condition to satisfy 
conditions of the "face area determination rule 1" and 
the "face area determination rule 2". 

The "face area determination rule 1" identifies 
one hair divided area group and one skin color divided 
area group as a set of areas and if the size ratio of the 
size Area_Hair of the one hair area group and the size 
Area_Skin of one skin color area group, i.e., 
Area_Hair/Area_Skin > 5 and Area_Hair/Area__Skin < 
1/5, determines as prospective face area. 

This "face determination rule 1" judges that this 
set of divided area group is most possible the face 
area if the size ratio of the sizes Area_Hair and 
Area__Skin is less than five times and more than 1/5 
times. 

The above judgement avoids the possibility of 



misjudging that all dark areas would be judged as hair 
area, such as the image having numerous divided 
areas with dark brightness level on the display screen 
PIC (e.g., in the case where there is a dark screen in 

5 the back), similarly, in the case where there exist a 
large quantity of skin color divided areas besides face 
skin, it is possible that the judgement of face area can 
not correctly be defined, and the above judgement of 
face area may not correctly be defined, and the 

10 above judgement thus avoids erroneous judgement. 

Furthermore, the face area determination rule 
2" defines that "if I (i, j) = hair area and I (i, j-1 ) = skin 
color area, Neighbor = Neighbor + 1. If Neighbor > 3, 
then hair area + skin color area is face". 

15 This "face area determination rule 2" judges a set 

of hair area group and skin color area group as the 
face if the upper side of the screen is hair divided area 
group (i.e., I (i, j)) and the lower side is skin color area 
group (i.e., I (i, j-1)) and under the condition that these 

20 hair divided area group and skin color divided area 
group are in contract with at least three divided areas. 

According to this "face area determination rule 
2", a pair of divided area groups, i.e., a set of hair area 
group HR1 and skin color area group FC1 is in contact 

25 with 8 divided areas and satisfies this rule 2, and 
therefore, it is judged as a face. 

On the other hand, because a pair of divided area 
groups, i.e., a set of hair area group HR2 and skin col- 
or area group FC2 is not in contact with any areas and 

30 does not satisfy the condition of rule 2, it is excluded 
from the judgement of face area. 

Furthermore, the tracking signal processing cir- 
cuit 16 proceeds to the step SP6 and judges whether 
the process of steps SP2 - SP5 has been repeated 

35 twice or not When the first process is finished, it pro- 
ceeds to the step SP7 and changes the divided num- 
ber divided at the step SP1 from 32 x 32 to 16 x 16 
and extracts prospective hair area and prospective 
skin color area and returns to the step SP2. 

40 Thus, the tracking signal processing circuit 16 

executes steps SP2 - SP3 - SP4 - SP5 on the divided 
areas- of 1 6 x 16, and thus improve the probability of 
face recognition by executing the process on roughly 
divided areas than the process on 32 x 32 divided. 

45 In this connection, there are cases where the 

face object which could not be recognized at the time 
of 32 x 32 divided process can be recognized by proc- 
essing on the roughly divided areas. For instance, it 
is especially profitable in case of treating a zoom uped 

so large face in the display screen PIC. 

When an affirmative result is obtained at the step 
SP6, it means that the image processing has been 
finished twice and the tracking signal processing cir- 
cuit 16 proceeds to the step SP8 and when more than 

55 two faces are extracted from one display screen PIC, 
it executes the process so as to select the face area 
near the center of display screen PIC as an object at 
the time when the record pause button is released, 
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and then at the step SP9, it supplies tracking control 
signal S5 in order that the centroid of face will be the 
center of the measuring frame, i.e., the display screen 
PIC, to the panning driving motor 12 and the tilting 
driving motor 13. 

Thus, the tracking signal processing circuit 16 
terminates a cycle of the automatic tracking process 
and returns to the step SP1 mentioned above to enter 
the following automatic tracking cycle. With this ar- 
rangement, since it is used as quantity of a character- 
istic a hue signal which is hard to be affected by an 
intensity and a direction of light source and the most 
effectively distinguishes the characteristic of the ob- 
ject on the display screen, an automonous target 
tracking apparatus which is hard to be affected by the 
background and having resistance to the movement 
of the hands can be obtained. 

In this connection, in the case where the charac- 
teristic quantity of the object is processed based on 
the brightness signal as before, if the lightness of the 
display screen PIC brightens up extremely or be- 
comes dimmer, as described above regarding the 
HLS color coordinate system in Fig. 2, it has been dif- 
ficult to grasp the image information of the object. 
However, in the case of the above embodiment, even 
through the lightness becomes extremely brighter or 
dimmer, the characteristic quantity of the object can- 
not be lost. 

Also, according to the foregoing construction, 
since the peak value has not been used as the char- 
acteristic quantity, it is resistant to noises and since 
the human face model which is not affected by 
changes of the coordinate on the screen is used with- 
out using the motion vector, the system which has re- 
sistance to the movement with hand of camera will be 
constructed. 

Furthermore, according to the foregoing con- 
struction, since it holds the human face model inside, 
the user does not have to initialize the object and 
there is no need for setting up the measuring frame 
and if there is a background in the measuring frame, 
the phenomenon that a center of a picture would be 
pulled by this cannot be occurred. 

Fig. 8 illustrates the other embodiment according 
to the present invention and as shown, the identical 
reference numerals are given with corresponding 
parts of Fig. 1, brightness signal Y. color difference 
signal R-Y and B-Y obtained from the digital camera 
processing circuit 7 will be given to the screen reduc- 
tion circuit 32 via a low pass filter 31. The screen re- 
duction circuit 32 executes the process to divide the 
display screen PIC into divided areas 32 x 32 or 16 x 
16 by means of screen reduction control signal S21 
outputted from the tracking signal processing circuit 
1 6 and brightness signal Y and color difference signal 
R-Y and B-Y of each pixel outputted from the digital 
camera processing circuit 7. 

Accordingly, hue signal HUE and saturation sig- 



nal SAT will be supplied to the image memory 15 by 
giving color difference signals R-Y and B-Y corre- 
sponding to 32 x 32 areas or 16 x 16 areas from the 
screen reduction circuit 32 to the saturation/hue de- 
5 tection circuit 1 4 as target tracking detection sig nal S4 
and simultaneously, brightness signal Y will be sup- 
plied directly to the image memory 1 5 as target track- 
ing detection signal. 

In this case, the image memory 1 5 has a memory 
w area for 32 x 32 images, therefore, in the case of Fig. 
1 it is necessary to have memory capacity for one 
field of pixel, on the other hand in the case of Fig. 8, 
it can be simplified further. 

With this arrangement, hue signal HUE and sat- 
15 uration signal SAT will be outputted as signals accu- 
mulated in 32 x 32 pixels or 16 x 16 pixels, and thus 
the hair detection circuit 20 and skin color detection 
circuit 19 detect divided areas 32 x 32 or 16 x 16 
whether they are the hair areas or skin color areas re- 
20 spectively and supply the detection result to the track- 
ing signal processing circuit 16 as hair area detection 
output S22 and skin color area detection output S23. 

According to the construction of Fig. 8, since it is 
so arranged that the saturation/hue detection circuit 
25 1 4, the image memory 1 5, the hair detection circuit 20 
and the skin color detection circuit 19 will be process- 
ed after the image signal for one screen to be output- 
ted from the digital camera processing circuit 7 is re- 
duced as image information of 32 x 32 divided areas 
30 or 1 6 x 1 6 divided areas at the screen reduction circuit 
1 9, the circuit construction of said processing circuits 
can be further simplified. 

In the case where the autonomous target track- 
ing apparatus ATT composed of Fig. 1 or Fig. 8 is ap- 
35 plied to an auto focus adjusting system, since the auto 
focus adjusting system tracks in order that the lens 
block unit 1 can set the measuring frame followed by 
the display screen PIC corresponding to the change 
of the face centroid , an automatic focus adjusting sys- 
40 tern which can focus the lens 2 on the centroid posi- 
tion of a face can be obtained. 

Also, in the case of applying the automonous tar- 
get tracking apparatus of the above mentioned con- 
struction to an automatic iris adjusting system, since 
45 it can be so arranged that the lens block unit 1 can 
set the measuring frame on the centroid of the face 
followed by the display screen PIC and the centroid 
of the face can be always light measured, it becomes 
unnecessary to correct the counter light as the auto 
50 iris adjusting system and simultaneously, the bright- 
ness level of the object can be set automatically to t he 
optimum value. 

Furthermore, in the case of applying the auton- 
omous target tracking apparatus to an automatic 
55 white balance adjusting system, since the lens block 
unit 1 sets the measuring frame to track the centroid 
of the face followed by the display screen PIC, the 
auto white balance adjusting system can adjust colors 
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of overall screen by adjusting to get optimal skin color 
or the face. And as a result the auto white balance ad- 
justing system as a whole can adjust the auto skin col- 
or balance (ASB) and thus, as compared with the pre- 
viously proposed auto white balance adjusting appa- 5 
ratus which adjusts because the integrated result of 
the entire screen is white, even in the case where the 
environment of the object is over exposed or under 
exposed, the display screen having excellent color 
balance can be obtained. 10 

Furthermore, the embodiment described above 
has dealt with the case of executing absorption, com- 
bination and determination processes on the group of 
hair divided areas and the group of skin color divided 
areas by setting the measuring frame on the overall 15 
display screen PIC. However, the same effects as 
those of the embodiment described above can be ob- 
tained if the measuring frame is set in the area select- 
ed according to demands in the display screen PIC. 

Moreover, the embodiment described above has 20 
dealt with the case of shifting the field of vision of the 
lens block 1 corresponding to the movement of the 
object by panning and/or tilting and thus the desired 
object image will be entered in the measuring frame. 
However, the same effects as those of the embodi- 25 
ment described above can be obtained by shifting the 
measuring frame for the image in the display screen 
PIC without panning or tilting. 

It will be apparent to those skilled in the art that 
various changes and modifications may be made to 30 
the specific embodiments described above without 
departing from the scope of the invention. 



Claims 35 

1 . A video camera system comprising: 

area dividing means (16) for dividing a dis- 
play image to be shown by image output signals 
output from a lens Nock unit (1 ) into divided areas 40 
(AR) of prescribed numbers; 

skin color area extracting means (1 6) for 
extracting skin color divided areas (ARB) wherein 
hue signals are skin color part in all of said divid- 
ed areas; 45 

lightness area extracting means (16) for 
extracting lightness divided areas (ARA) wherein 
brightness signal are in prescribed luminance in 
all of said divided areas; 

absorbing means (1 6) for forming skin col- so 
or divided area groups and lightness divided area 
groups by absorbing blank divided areas around 
said skin color divided area and said lightness 
divided area; 

adjacent divided area extracting means 55 
(16) for extracting a set of skin color divided area 
group and lightness divided area group which are 
adjacent each other from said skin color divided 
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area groups and lightness divided area groups; 
and 

measuring frame setting means (16) for 
setting said measuring frame in order that said 
set of skin color divided area group and lightness 
divided area group can be entered into said 
measuring frame. 

2. A video camera system as claimed in claim 1, 
wherein said measuring frame setting means sets 
the image part of a set of said skin color divided 
area group and lightness divided area group with- 
in said measuring frame by moving said measur- 
ing frame on said display screen depending upon 
the extracted result of said adjacent divided area 
group extracting means. 

3. A video camera system as claimed in claim 1, 
wherein said measuring frame setting means sets 
the image part of a set of said skin color divided 
area group said lightness divided area group 
within said measuring frame by panning and/or 
tilting said lens block unit depending upon the ex- 
tracted result of said adjacent divided area group 
extracting means. 

4. A video camera system comprising: 

area dividing means (16) for dividing a dis- 
play image to be shown by image output signals 
output from the lens block unit (1) into divided 
areas (AR) of the prescribed numbers; 

skin color area extracting means (16) for 
extracting skin color divided areas (ARB) wherein 
hue signal is skin color part in all of said divided 
areas; 

hair area extracting means for extracting 
hair divided areas wherein brightness signal has 
hair lightness in all of said divided areas; 

absorbing means (16) for forming skin col- 
or divided area groups and hair divided area 
groups by absorbing blank divided areas around 
said skin color divided area and said hair divided 
area; 

adjacent divided area extracting means 
(16) for extracting a set of skin color divided area 
group and hair divided area group which are ad- 
jacent each other from said skin color divided 
area groups and hair divided area groups; 

face defining means forjudging said set of 
skin color divided area group and hair divided 
area group to be a face under the condition that 
said hair divided area group is adjacent to the up- 
per part of said skin color divided area group in 
said set of skin color divided area group and hair 
divided area group extracted in said adjacent div- 
ided area extracting means; and 

measuring frame setting means (16) for 
setting said measuring frame in order that said 
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face area is entered into said measuring frame. 

A video camera system as claimed in claim 4, 
wherein said measuring frame setting means (16) 
for setting said face area to enter in said measur- 5 
ing frame by panning and/or tilting said lens block 
unit (1) in order that the centroid of said face 
comes in the center of said display screen. 

A video camera system comprising: to 

area dividing means (16) for dividing a dis- 
play image to be shown by image output signals 
output from a lens block unit (1) into divided areas 
(AR) of the prescribed numbers; 

skin color area extracting means (16) for 15 
extracting skin color divided areas (ARB) wherein 
hue signal is skin color part in all of said divided 
areas; 

hair area extracting means for extracting 
hair divided areas wherein brightness signal has 20 
lightness of hair in all of said divided areas; 

absorbing means (16) for forming skin col- 
or divided area groups and hair divided area 
groups by absorbing blank divided areas around 
said skin color divided area and said hair divided 25 
area; 

adjacent divided area group extracting 
means (16) for extracting a set of skin color div- 
ided area group and hair divided area group 
which are adjacent each other from said skin col- 30 
or divided area groups and hair divided area 
groups; 

face defining means for judging said set of 
hair divided area group and skin color divided 
area group to be a face under the condition that 35 
the sizes of said hair divided area group and said 
skin color divided area group are calculated re- 
garding said set of skin color divided area group 
and hair divided area group extracted in said ad- 
jacent divided area group extracting means, the 40 
ratio of said sizes of said hair divided area group 
and said skin color divided area group is within 
the limit of the prescribed numbers, and said hair 
divided area group is adjacent the upper side of 
said skin color divided area group; and 45 

measuring frame setting means (16) for 
setting said measuring frame in order that said 
face area can be entered into said measuring 
frame. 

50 

7. An automonous target tracking method for auto- 
matically tracking an object displayed on a dis- 
play screen based on imaging output signals out- 
put from a lens block unit (1), said automonous 
target tracking method comprising the steps of: 55 

dividing a display image to be shown by 
said image output signals into divided areas (AR) 
of prescribed numbers; 



extracting skin color divided areas (ARB) 
wherein hue signal is skin color part in all of said 
divided areas; 

extracting lightness divided areas wherein 
brightness signal are in prescribed* luminance in 
all of said divided areas; 

absorbing block divided areas around said 
skin color divided area and said lightness divided 
area to form skin color divided area groups and 
lightness divided area groups; 

extracting a set of skin color divided area 
group and lightness divided area group which are 
adjacent each other from said skin color divided 
area groups and lightness divided area groups; 
and 

setting said measuring frame in order that 
said set of skin color divided area group and light- 
ness divided area group can be entered into said 
measuring frame. 

8. An automonous target tracking method for au- 
tomonous tracking an object displayed on a dis- 
play screen based on an imaging output signals 
output from a lens block unit (1 ), said automonous 
target tracking method comprising the steps of. 

dividing a display image to be shown by 
said image output signals into divided areas (AR) 
of prescribed numbers; 

extracting skin color divided areas (ARB) 
wherein hue signal is skin color part in all of said 
divided areas; 

extracting hair divided areas wherein 
brightness signal has hair lightness in all of said 
divided areas; 

absorbing blank divided areas around said 
skin color divided areas and said hair divided 
areas to form skin color divided area groups and 
hair divided area groups; 

extracting a set of skin color divided area 
group and hair divided area group which area ad- 
jacent each other from said skin color divided 
area groups and hair divided are groups; 

judging said set of skin color divided area 
group and hair divided area group to be a face un- 
der the condition that said hair divided area group 
is adjacent to the upper part of said skin color div- 
ided area group in said set of skin color divided 
area group and hair divided area group; and 

setting said measuring frame in order that 
said face area is entered into said measuring 
frame. 

9. An automonous target tracking method for auto- 
matically tracking an object displayed on a dis- 
play screen based on imaging output signals out- 
put from a lens block unit, said automonous target 
tracking method comprising the steps of: 

dividing a display image to be shown by 
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said image output signals wio divided areas (AR) 
of prescribed numbers; 

extracting skin color divided areas (ARB) 
wherein hue signal is skin color part in all of said 
divided areas; 5 

extracting hair divided areas wherein 
brightness signal has lightness of hair in all of said 
divided areas; 

absorbing blank divided areas around said 
skin color divided areas and said hair divided 10 
areas to form skin color divided area groups and 
hair divided area groups; 

extracting a set of skin color divided area 
group and hair divided area group which are ad- 
jacent each other from said skin color divided 15 
area groups and hair divided area groups; 

judging said set of hair divided area group 
and skin color divided area group to be a face un- 
der the condition that sizes of said hair divided 
area group and said skin color divided area group 20 
are calculated regarding said set of skin color div- 
ided area group and hair divided area group, the 
ratio of said sizes of said hair divided area group 
and said skin color divided area group is within 
the limit of the prescribed numbers, and said hair 25 
divided area group is adjacent upper side of said 
skin color divided area group: and 

setting said measuring frame in order that 
said face area can be entered into said measuring 
frame. 30 
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OIVIDE IMAGE (32x32) 
AND EXTRACT PROSPECTIVE 
HAIR AREA, PROSPECTIVE 
SKIN COLOR AREA. FN=FN+1 
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ABSORB EACH PROSPECTIVE AREA IN ACCORDANCE WITH 
FOLLOWING RULES: 

RULE 1; 

IF I (i+p, j+q) - PROSPECTIVE HAIR AREA 

(p.q - +1 OR -1 OR 0, PROVIDED, EXCEPT p=q=0), 

THEN I (i.j) = PROSPECTIVE HAIR AREA 

RULE 2; 

IF I (i+p, j+q) = PROSPECTIVE SKIN COLOR AREA 
(p,q = +1 OR -1 OR 0, PROVIDED, EXCEPT p=q=0), 
THEN I (i.j) = PROSPECTIVE SKIN COLOR AREA. 



COMBINATION OF HAIR AREA AND 
SKIN COLOR AREA 

COMBINE HAIR AREA AND FACE AREA IN ACCORDANCE WITH 
FOLLOWING RULES: 

RULE 1; t 
IF I (i.j) = PROSPECTIVE HAIR AREA AND I (i+p. j+q) 
- PROSPECTIVE HAIR AREA (p.q = +1 OR -1 OR 0, 
PROVIDED. EXCEPT p=q=0), THEN COMBINE I (i.j), 
I (i+p, j+q) AS HAIR AREA. 

RULE 2: 

IF I (i.j) = PROSPECTIVE SKIN COLOR AREA ANO 
I (i+p, j+q) = PROSPECTIVE SKIN COLOR AREA 
(p.q = +1 OR -1 OR 0, PROVIDED, EXCEPT p=q=0). 
THEN COMBINE I (i.j). I (i+p. j+q) AS SKIN COLOR 
AREA. 
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DETERMINATION 
OF FACE AREA 

IF FOLLOWING 2 RULES ARE SATISFIED, THEN JUDGE 
AS FACE AREA: 

RULE 1; 

IF AREA HAIR/AREA SKIN < 5 AND 

AREA HATR/AREA SKTN > 1/5. THEN PROSPECTIVE FACE AREA. 



IF I (i. j) = HAIR AREA AND I (i, = SKIN COLOR 

AREA, THEN NEIGHBOR = NEIGHBOR + 1. 

IF NEIGHBOR > 3, THEN HAIR AREA + SKIN COLOR AREA = 
FACE. 



DIVIDE IMAGE INTO DIVIDED AREA 16x16 
AT SP1 AND EXTRACT PROSPECTIVE HAIR 
AREA AND SKIN COLOR AREA, THEN 
REPEAT FROM SP2. 
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(g) Video camera with colour-based target tracking system. 

(57) A video camera system comprising a teaching 
apparatus in which the field of view of a video 
camera can track the center of an object such a 
human face. 

A display screen is divided into a lightness 
divided area in which a brightness signal has 
the prescribed luminance, and a skin color 
divided area in which a hue signal is skin color. 
Depending upon the extraction result, the posi- 
tion on the screen of the object is detected. A 
measuring frame which is not be affected by 
intensity and direction of the light source can 
then be set utilizing the hue signal. 
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